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COHERENT OPTICAL DETECTION AND SIGNAL 
PROCESSING METHOD AND SYSTEM 

BACKGROUND OF THE INVENTION 
RELATED APPLICATIONS 

[0001] This utility application claims the priority date of Provisional 
Patent Application Serial No. 60/432500 by Michael G. Taylor, filed December 
11, 2002 and Provisional Patent Application No. 60/445,742 by Michael G. 
Taylor, filed February 7, 2003 and is hereby incorporated by reference. 

BACKGROUND OF THE INVENTION 

Field of the Invention : 
[0002] This invention relates to optical transmissions over a fiber optic 
cable. Specifically, the present invention relates to a system and method of 
coherent detection of optical signals by utilizing digital signal processing to 
recover signals. 
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Description of the Related Art: 
[0003] The use of optical fiber cables for the transmission of 
information was introduced several years ago. Recently, with the hunger by 
users for the rapid transmission of large amounts of information, the utility of 
the transmission of optical signals is particularly evident, [see "optical 
Communication Systems" by J. Gowar (Gowar) and "Fiber-optic 
Communication Systems" by G. Agrawal (Agrawal 2)]. The transmission of 
this information typically takes the form of binary digital signs (i.e., logical T's 
and "0"s. In addition, fiber optics is utilized to transport analog signs, such as 
cable television signals. 

[0004] In the 1990s, optical amplifiers were deployed in telephonic and 
cable television networks. Typically, erbium doped fiber amplifiers (EDFAs) 
were employed. The amplifiers amplify the optical signals and overcome the 
loss of a signal transmitted over the fiber without the need to detect and 
retransmit the signals. In addition, the 1990s saw the introduction of 
wavelength division multiplexing (WDM) on a commercial level, which 
increased the information carrying capacity of the fiber by transmitting several 
different wavelengths in parallel. During WDM, different wavelengths 
originate and terminate at the same place, but in some cases, wavelengths 
are added or dropped in route to a destination. 

[0005] With optical signal transmission systems, each system has a 
transmitter to emit light modulated with information through the fiber optics 
cable and a receiver, which detects the light and recovers the information. 
The transmission unit contains a light source, usually a single longitudinal 
mode semiconductor laser. Information is imposed on the light by direct 
modulation of the laser current, or by external modulation (by applying a 
voltage to a modulator component that follows the laser signal). The receiver 
utilizes a photodetector, which converts light into an electric current. 
[0006] There are currently two ways to detect the light, direct detection 
and coherent detection. Existing transmission systems all use direct detection 
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techniques for detecting the light Although coherent detection techniques are 
more complex, they do offer some distinct advantages. One of the primary 
advantages is that it effectively provides signal gain. However, the EDFA 
offered the same advantage, and it was found to a more cost-effective 
solution. 

[0007] Current transmission systems impose information on the 
amplitude (or intensity/power) of the signal. The light is switched on to 
transmit a "1" and off to transmit a "0." In the case of direct detection 
techniques, the photodetector is presented with the on-off modulated light, 
and consequently the current flowing through it is a replica of the optical 
power. After amplification, the electrical signal is passed to a decision circuit, 
which compares it to a reference value. The decision circuit then outputs an 
unambiguous "1" or "0." 

[0008] There are many kinds of on-off modulation formats. The 
simplest one is unchirped non-retum-to-zero (unchirped NRZ), where the 
optical power and the phase of the optical wave are kept constant during the 
transition between a pair of consecutive "1" symbols. Other transmission 
formats are chirped NRZ, return-to-zero (RZ) [see "Comparison between NRZ 
and RZ signal formats for in-line amplifier transmission in the zero-dispersion 
regime" by Matsuda (Matsuda)], carrier suppressed RZ [see " 100 GHZ- 
spaced 8x43 Gbit/s DWDM unrepeatered transmission over 163 km using 
duobinary-carrier-suppressed return-to-zero format" by Y. Miyamoto et al. 
(Miyamoto)] and phase shaped binary transmission (PSBT) [see " The phase- 
shaped binary transmission (PSBT): a new technique to transmit far beyond 
the chromatic dispersion limit" by D. Penninckx et al. (Penninckx)]. 
[0009] There is also another class of modulation formats where 
information is encoded on the phase of the optical signal, such as optical 
differential phase shift keying (oDPSK). A photodetector does not respond to 
changes in the phase of the light falling on it, so a passive component called a 
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discriminator is used before the photodetector. The discriminator converts the 
changes in phase into changes in power, which the photodetector may detect. 
[0010] As discussed above, the photodetector does not respond to the 
phase of an optical wave. If two wavelengths are input to the photodetector, 
for example, the photodetector does not distinguish between the two 
wavelengths. The WDM systems utilize passive optical filter components to 
separate out the different wavelength channels at the receiver terminal, so 
each photodetector detects only one channel. This approach places a limit on 
how close the channels may be spaced, primarily based upon the optical 
filter's ability to pass one channel and reject its neighbors. 
[0011] The coherent detection techniques treat the optical wave in a 
manner similar to radio wave reception by inherently selecting one 
wavelength and responding to its amplitude and phase. FIG. 1A illustrates a 
simplified block diagram of a single ended basic coherent receiver 16 in an 
existing fiber optics system. FIG. 1B illustrates a simplified block diagram of a 
coherent receiver 21 using balanced detection with two photodetectors 27 and 
29 in an existing fiber optics system. An incoming signal 18 is combined with 
light from a local oscillator (LO) 20, which has close to the same state of 
polarization (SOP) and the exact or very similar wavelength. When the 
combined signals are detected, the photocurrent contains a component at a 
frequency which is the difference between the signal and the local oscillator 
optical frequencies. This difference frequency component contains all the 
information (amplitude and phase) that is on the optical signal. Because the 
new carrier frequency is much lower, typically a few GHz instead of 200THz, 
all information on the signal can be recovered using standard radio 
demodulation methods. Coherent receivers see only signals close in 
wavelength to the local oscillator. Therefore, tuning the LO wavelength 
provides the functionality of a built-in tunable filter. 

[0012] The coherent detection process may be explained with several 
mathematic equations. The following description utilizes complex notation for 
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sinusoids that are summarized in Appendix A. The electric field of the signal 
may be written as: 



Re 



where E s (t) is the slowly varying envelope containing the information 

encoded on amplitude and phase of the optical signal. Similarly, the electric 
field of the local oscillator may be described as: 

where E L0 is a constant for a local oscillator. The electric field of the light 

arriving at the photodetector 29 in the top branch of FIG. 1B (or the 
photodetector 24 in FIG. 1A) is the sum of the two electric fields: 

and the optical power is: 

p x =e;e, =(£/(/>^ +£ W v^](E I (/y^ + 

P^ =\E s (tf +\E L0 \ 2 + 2Re[£ s (^ 0 *e^-^>] (1) 

[0013] In the case of single ended detection, only one output of the 

combiner is used. \E L0 \ 2 is constant with time. |£ s (0| 2 is relatively small, 

given that the local oscillator power is much larger than the signal power. In 
addition, for the phase shift keying (PSK) and frequency shift keying (FSK) 

modulation formats \E s (t] 2 is constant with time. The dominant term in 
equation 1 is the beat term K^E s {f)E L0 \^ s ' mu ^\ 

[0014] The output of the lower branch is the difference of the two 
electric fields, and the optical power is: 

Pi =Mf + \ £ lo\ 2 -2Re[E s (t)E L oe^-""> >] (2) 
The other mode of detection is balanced detection, where the electrical 
circuitry after the photodetectors evaluates the difference in photocurrent 
between the two detectors: 
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P x -P 2 ^AK^E s {t)E LO \^^o)t\ (3) 

Balanced detection produces the beat term directly. With balanced detection, 
there is no need for the constraint that the local oscillator power should be 
greater than the signal power. It has the additional advantage that noise on 
the local oscillator is subtracted out. 

[0015] The following equations refer to the beat term directly, and it is 
assumed that this term is obtained by single ended detection (without the 
contribution of other terms) or by balanced detection. 
[0016] There are two modes of coherent detection: homodyne and 
heterodyne. With homodyne detection, the frequency difference between the 
signal and the local oscillator is zero. The local oscillator laser has to be 
phase locked to the incoming signal in order to achieve this. For homodyne 

detection the term e^ m '~ mL °^ is 1, and the beat term becomes 

Rs[E s (t)E L o\ 

[0017] For the binary phase shift keying (BPSK) modulation format, 
E s {t) takes on the value 1 or -1 depending on whether a logical "1" or "0" was 

transmitted, and the decision circuit can simply act on the beat term directly. 
Homodyne detection requires that the bandwidth of the photodetector and the 
subsequent components be close to the bit rate. In addition, homodyne 
detection gives a better sensitivity than any other way of detecting the signal. 
Also homodyne detection has an inherent ultranarrow optical filtering 
capability, in that all regions of the optical spectrum, which are more that the 
detector bandwidth away from the local oscillator, are rejected. This feature 
means that homodyne detection can support a higher density of WDM 
channels than by using passive optical filters for WDM demultiplexing. The 
homodyne detection method has the disadvantage that the local oscillator 
must be phase locked to the signal. The local oscillator and signal lasers 
must be narrow linewidth lasers, such as external cavity semiconductor 
lasers, which are typically more expensive than the distributed feedback 
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(DFB) laser. Additionally, some polarization management methods do not 
work with homodyne detection. 

[0018] With heterodyne detection, there is a finite difference in optical 
frequency between the signal and local oscillator. All the amplitude and 
phase information on the signal appears on a carrier at angular frequency 
(o) s -o) LO ) % known as the intermediate frequency (IF), which can be detected 

using standard radio detection methods (e.g., synchronous detection, 
envelope detection or differential detection). Heterodyne detection has the 
advantage that the local oscillator does not need to be phase locked, and a 
DFB laser can be used for the LO and the signal lasers. Also it is possible to 
employ signal processing in the IF to compensate for chromatic dispersion, 
which is considered impossible to do with homodyne detection by existing 
techniques. The heterodyne detection processes suffer from the 
disadvantage that the difference frequency must be at least equal to half the 
optical spectral width of the signal, about 0.75 times the symbol rate, to avoid 
a penalty from self-imaging, which requires the bandwidth of the 
photodetector to be at least 1.5 times the symbol rate. The sensitivity of 
heterodyne detection is 3dB worse than homodyne detection. In addition, for 
heterodyning to work, there must be an empty region in the optical spectrum 
adjacent to the signal being detected, which constrains the density at which 
WDM channels can be packed. 

[0019] A system and method is needed which includes all the 
advantages of homodyne detection in conjunction with all of the advantages 
of heterodyne detection. In addition, a system and method is needed which 
can vary the chromatic dispersion compensation and subtract cross talk from 
other WDM channels. 

[0020] Thus, it would be a distinct advantage to have a system and 
method which incorporates coherent detection of optical signals with digital 
signal processing to recover a signal when a local oscillator is not phase 
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locked to the signal It is an object of the present invention to provide such a 
system and method. 

SUMMARY OF THE INVENTION 
[0021] In one aspect, the present invention is a coherent optical 
detection system receiving an optical signal in a fiber optics network. The 
system includes a local oscillator emitting light and a phase diverse hybrid for 
mixing the incoming optical signal with the local oscillator light in two outputs. 
The phase diverse hybrid creates two replicas of the incoming optical signal 
and two replicas of the light from the local oscillator, and then combines one 
replica of the received signal and one replica of the local oscillator light in one 
of the outputs, and the other two replicas in the other output. The state of 
polarization of the optical signal and the light from the local oscillator are close 
to aligned in each of the two outputs. However, the local oscillator does not 
have to be phase locked to the incoming optical signal. In addition, the phase 
relationship between the optical signal and the local oscillator light in a first 
output of the two outputs is different by approximately 90 degrees compared 
to the phase relationship between the local oscillator light and the optical 
signal in a second output of the two outputs. The system also includes at 
least two photodetectors connected to the two outputs of the phase diverse 
hybrid. The two photodetectors receive optical signals from the two outputs 
and convert the optical signals to electrical signals. The received electrical 
signals are processed to provide a complex representation of the envelope of 
the electric field of the incoming optical signal. In addition, this coherent 
detection system may be used when the difference frequency is less that 
twice the symbol rate. 

[0022] In another aspect, the present invention is a coherent optical 
detection system receiving an optical signal in a fiber optics network. This 
system includes a local oscillator emitting light and a polarization diversity 
hybrid for generating at least two replicas of the incoming optical signal and 
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local oscillator light. The polarization diversity hybrid combines the incoming 
optical signal and the local oscillator light into at least two outputs. The state 
of polarization of the first replica of the optical signal relative to the local 
oscillator light at the first output of the two outputs is close to orthogonal to the 
state of polarization of the second replica of the optical signal relative to the 
local oscillator light at the second output of the two outputs. The system also 
includes at least two photodetectors connected to the two outputs of the 
polarization diversity hybrid. The two photodetectors receive optical signals 
from the two outputs. In addition, the system includes at least two A/D 
converters to receive electrical signals from the two photodetectors. The two 
A/D converters digitize the received electrical signals. A digital signal 
processor is utilized to perform computations on digitized values converted by 
the two A/D converters to obtain information carried by the incoming optical 
signal. The system enables information on the incoming optical signal to be 
recovered irrespective of the state of polarization of the incoming signal. 
[0023] In another aspect, the present invention is a coherent optical 
detection system receiving an optical signal in a fiber optics network. The 
optical signal comprises two different optical signals that have been 
polarization multiplexed together. The coherent optical detection system 
includes a local oscillator emitting light and a polarization diversity hybrid for 
generating at least two replicas of the incoming optical signal and local 
oscillator light. The polarization diversity hybrid combines the incoming optical 
signal and the local oscillator light into at least two outputs. The state of 
polarization of the first replica of the optical signal relative to the local 
oscillator light at the first output of the two outputs is close to orthogonal to the 
state of polarization of the second replica of the optical signal relative to the 
local oscillator light at the second output of the two outputs. The system also 
includes at least two photodetectors connected to the two outputs of the 
polarization diversity hybrid. The two photodetectors receive optical signals 
from the two outputs. In addition, the system includes at least two A/D 
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converters to receive electrical signals from the two photodetectors. The two 
A/D converters digitize the received electrical signals. A digital signal 
processor is utilized to perform computations on digitized values converted by 
the two A/D converters to obtain information carried by each of the 
polarization multiplexed tributaries comprising the incoming optical signal. 
[0024] In another aspect, the present invention is a coherent optical 
detection system receiving an optical signal in a fiber optics network. The 
system includes a local oscillator emitting light and an optical mixing hybrid for 
combining the incoming optical signal and the local oscillator light into at least 
one output. A photodetector is connected to the output of the optical mixing 
hybrid and receives optical signals from the output. The system also includes 
an A/D converter to receive electrical signals from the photodetector and 
digitize the electrical signals. A digital signal processor performs 
computations on digitized values converted by the AID converter. The digital 
signal processor estimates a complex envelope of an electric field of the 
incoming optical signal and performs a signal processing operation on a 
plurality of samples over time of the complex envelope of the electric field. 
[0025] In another aspect, the present invention is a coherent optical 
detection system receiving an optical signal having a plurality of wave division 
multiplexed (WDM) channels. The system includes at least one local 
oscillator emitting light and an optical mixing hybrid for combining the 
incoming optical signal and the local oscillator light into at least one output. 
The system also includes a photodetector connected to the output of the 
optical mixing hybrid to receive optical signals from the output. An A/D 
converter receives electrical signals from the photodetector and digitizes the 
electrical signals. A digital signal processor performs computations on 
digitized values converted by the A/D converter. The digital signal processor 
estimates information carried on one of the plurality of WDM channels while 
accounting for the other WDM channels and subtracting crosstalk from the 
plurality of WDM channels. 
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[0026] In still another aspect, the present invention is a coherent optical 
detection system receiving an optical signal in a fiber optics network. The 
system includes a local oscillator emitting light and an optical mixing hybrid. 
The local oscillator does not have to be phase locked to the incoming optical 
signal. Within the optical mixing hybrid at least four replicas of the incoming 
optical signal and four replicas of the light from the local oscillator are 
generated, and replicas of the incoming signal and local oscillator are 
combined into at least four outputs. The phase or state of polarization of the 
local oscillator light relative to the incoming signal at the first output is different 
from the phase or state of polarization at the other three outputs, and similarly 
for the second, third and fourth outputs. Four photodetectors are connected 
to the four outputs of the mixing hybrid and receive optical signals from the 
four outputs and convert them to electrical signals. In addition, four A/D 
converters digitize the received electrical signals. A digital signal processor 
performs computation on digitized values from the four A/D converters to 
obtain information carried by the received signal without limitation to the state 
of polarization of the received optical signal. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0027] FIG. 1A (Prior Art) illustrates a simplified block diagram of a 
single ended basic coherent receiver 16 in an existing fiber optics system; 
[0028] FIG. 1B (Prior Art) illustrates a simplified block diagram of a 
coherent receiver 21 using balanced detection in an existing fiber optics 
system; 

[0029] FIG. 2A is a simplified block diagram illustrating the components 
of a generic analog signal processor (ASP); 

[0030] FIG. 2B is a simplified block diagram illustrating the components 
of a generic digital signal processor; 
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[0031] FIG. 3A illustrates a simplified block diagram of a single ended 
quadrature sampling receiver in the preferred embodiment of the present 
invention; 

[0032] FIG. 3B illustrates a simplified block diagram of a balanced 
quadrature sampling receiver in a first alternate embodiment of the present 
invention; 

[0033] FIG. 4A is a simplified block diagram of an existing configuration 
for polarization diversity detection; 

[0034] FIG. 4B is a simplified block diagram of a configuration using 
two LO lasers for polarization diversity detection; 

[0035] FIG. 5 is a simplified block diagram illustrating a polarization 
tracking system 131 in the preferred embodiment of the present invention; 
[0036] FIG. 6 is a simplified block diagram illustrating quadrature 
sampling utilizing analog signal processing in a first alternate embodiment of 
the present invention; 

[0037] FIG. 7A is a simplified block diagram illustrating a subsystem 
that contributes MPI containing a pair of optical amplifiers connected in 
parallel; and 

[0038] FIG. 7B is a graphical representation of an associated gain 
spectrum for FIG. 7A. 

DESCRIPTION OF THE INVENTION 
[0039] In the present invention, digital signal processing (DSP) is 
employed to recover optical signals. Mr. Richard Lyons in "Understanding 
Digital Signal Processing" discloses the basic principles behind DSP. FIG. 2A 
is a simplified block diagram illustrating the components of a generic analog 
signal processor (ASP) 30. A signal processor is a unit, which takes in a 
signal, typically a voltage vs. time, and performs a predictable transformation 
on it. This transformation may be described by a mathematical function. The 



12 



5034-0001 
PATENT 

ASP 30 transforms an input signal voltage 32 into an output signal voltage 34. 
The ASP may utilize capacitors, resistors, inductors, transistors, etc. to 
accomplish this transformation. FIG. 2B is a simplified block diagram 
illustrating the components of a generic digital signal processor 36. An input 
signal 38 is digitized by an analog to digital (A/D) converter 40 that is 
converted into a sequence of numbers. Each number represents a discrete 
time sample. A core processor 42 uses the input numerical values to 
compute the required output numerical values, according to a mathematical 
formula that produces the required signal processing behavior. The output 
values are then converted into a continuous voltage vs. time output signal 46 
by a digital to analog (D/A) converter 44. Alternatively, for applications in a 
digital signal receiver, the analog output of the DSP may go into a decision 
circuit to produce a digital output. In such a situation, the digital processing 
core may perform the decision operation and output the result, in which case 
the D/A 44 is not needed. 

[0040] There are disclosures on equalization (signal processing) to 
correct for fiber optic transmission impairments [see "Electronic PMD 
mitigation-from linear equalization to maximum-likelihood detection" by H. 
Bulow et al. (Bulow 1) and "Electronic equalization of transmission 
impairments" by H. Bulow et al. (Bulow 2)]. In this disclosure, direct detection 
is used and the equalizer is located after the detector. Most of the existing 
works are on analog signal processing, although digital signal processing has 
been proposed and simulated [see "Design of near optimum electrical 
equalizers for optical transmission in the presence of PMD" by H. F. 
Haunstein et al. (Haunstein) and "DSP-Based Equalization for Optical 
Channels-Feasibility of VLSI Implementation" by O. Agazzi (Agazzi)]. The 
present invention uses DSP to correct transmission impairments, but is used 
in conjunction with coherent detection. 
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Quadrature Sampling 
How Quadrature Sampling Works 

[0041] The present invention provides for recovery of information on an 
optical signal using a local oscillator which is not phase locked to the signal, 
and which may have an optical frequency arbitrarily close to that of the signal. 
FIG. 3A illustrates a simplified block diagram of a single ended quadrature 
sampling receiver 50 in the preferred embodiment of the present invention. A 
90° hybrid passive unit 54 is depicted within the dotted box. The method of 
recovering a digital representation of the complex envelope of the signal 
electric field is known herein in the present invention as quadrature sampling. 
The 90° hybrid passive unit mixes the signal with a local oscillator 56 in two 
paths, such that the phase difference between the signal and LO 56 in one 
path differs from the phase difference in the other path by about 90°. There 
are many ways to make the 90° hybrid. As illustrated in FIG. 3A, the 90° 
hybrid passive unit splits both the signal and LO and then combines in each 
output arm a replica of the signal with a replica of the LO. However, there is 
an extra path length in one arm of the LO splitter to apply the phase shift. The 

electric field of the local oscillator may be written as Re E LO e ia}LO * \ for the top 

path and Re[/£ zo e'^'] for the bottom path, while the signal is Re^/)?'^'] 

in both paths. After mixing the LO with the signal, the beat term for the top 
path is, following equation 2 or 3: 

beat term l = Rc[e s (t)E L0 *e^ ^ Lo)t j (4) 

and for the lower arm: 

beat term 2 = Re[- iE s {t)E L Q e^'^ J 

beat term 2 = Iu\[e s {t)E L0 * e^'^ J (5) 

[0042] Two A/D converters 58 and 60, utilized after the photodetectors 
62 and 64, in the two paths convert the photocurrents proportional to these 
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two beat terms into a sequence of numerical values versus time. The digital 
signal processor unit 36 accepts inputs from the A/D converters in both paths. 
The DSP is capable of doing computations on complex numbers. The DSP is 
also able to calculate from its inputs, the complex envelope of the signal 
electric field, E s (t), using the following formula: 

e -i{o) s -o) IO )t 

E s( t ) = ; — [( beot term 0 + ^ eat term 2 )] ( 6 ) 

E LO 

Equation 6 is obtained by combining equations 4 and 5. The equation may be 
written in terms of only real quantities as: 

Re[£ 5 (f)] = 7— — | [cos((<y, -co IO )t -<fi)(beat term l)+sin((<s; v -a) l0 )t -<f>\beat term 2)] 

\ E LO\ 

(7a) 

M^(0] = TTr-i[- sin ((^5 -co LO }-4>Xbeat term\) + cos({o) s - a> L0 )t - </>\beat term!)] 

\ E LO\ 

(7b) 

<f> is the argument (phase angle) of E L0 . This method of recovering a digital 
representation of a complex signal is known as quadrature sampling. This 
method may also be referred to as a heterodyne detection followed by 
synchronous demodulation using a complex local oscillator and digital phase 
estimation. Although quadrature sampling is used in radio communications, it 
has never been applied to the detection of an optical signal before. By 
combining sampled values from the two paths of the 90° hybrid passive unit 
into complex numbers, it is possible to perform heterodyne detection without 
problems from self-imaging even when the IF is much lower than the bit rate. 
Equation 6 assumes that the two beat terms are effectively sampled at the 
same instant. If the path lengths are not equal from the signal splitter to the 
two A/Ds, then this will cause timing skew. The DSP can compensate for the 
skew by using an elastic buffer store at one of its inputs. 



15 



5034-0001 
PATENT 



[0043] The rotating phasor in equation 6, e'^"^ I E LO \ contains 
(co s -co L0 )i-<j> , the phase of the signal with respect to the LO, which is not 
provided directly to the DSP and must be calculated by it from beat term 1 and 
beat term!. Only when the estimate of (co s -co LO )t-0 is correct continuously 

over time (when the phase estimation algorithm is locked) may the data be 
recovered with a low bit error rate. After locking has occurred, the phase term 
wanders because of the finite linewidth of the signal and LO lasers over a time 
of typically many bit periods. The phase estimation algorithm must then track 
this phase wander. There are many types of phase estimation algorithm that 
can be implemented within the DSP, as described in "Digital Communications" 
by John G. Proakis (Proakis). If the modulation format of the incoming signal 
contains a pilot carrier then a digital phase locked loop (PLL) or an open loop 
phase estimation algorithm can be applied to (beat term \)+i(beat term!) 
directly. An example of an open loop phase estimation algorithm is taking the 
arctangent of the ratio (beat term 2)/(beat term l) followed by a low pass filter 
function. When the signal's modulation format is such that it contains no 
carrier then (beat term \)+i(beat term 2) must first be processed by a single line 
generation function, such as a squaring function (or power law function in the 
case of high order PSK), or a decision directed multiplication, or in a Costas 
loop which combines the single line generation function with the PLL. An 
alternative method of estimating the phase is to make use of known 
sequences of symbols that are repeated in the transmitted signal every time 
interval r segi but this kind of method is useful only when the frequency 

difference between the signal and local oscillator is small, less than M2r seq . 

[0044] Receivers of digital information generally employ a decision 
circuit which is clocked (makes a decision) at a frequency exactly equal to the 
symbol frequency. A clock recovery circuit is used to derive a clock signal 
from the incoming data stream. The clock recovery circuit may be 
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accomplished within a separate analog circuit separate from the DSP or, in an 
alternate embodiment of the present invention, located in the same package 
as the DSP logic gates. In the present invention, the sample rate of the AID 
converters may be set equal to the symbol frequency by the clock recovery 
circuit. Alternatively, the A/D converters may be clocked faster than the 
symbol frequency by a clock that is asynchronous with the symbol frequency 
(usually at least 2X faster), and the sample values at the center of the symbol 
may be estimated by the DSP using an interpolation method. 
[0045] Like any optical receiver, the coherent receiver will, in general, 
have a finite baseband bandwidth, usually associated with a lowpass 
response. This finite bandwidth serves the useful purpose of limiting the 
amount of noise seen by the receiver, and when the receiver is used to detect 
one of many WDM channels, the finite baseband bandwidth also determines 
the ability to reject neighboring channels. Filter components can be purposely 
inserted in the receiver to limit the bandwidth, or the natural response (e.g., of 
the photodetector) can be exploited, or a digital filter function implemented 
within the DSP can be used. One kind of purposely inserted filter is the 
integrate-and-dump filter, which has the property that a neighboring WDM 
channel is completely rejected, provided it is centered on an orthogonal 
frequency. For most other filter responses that might be used, the concept of 
a set of orthogonal frequencies does not apply, and the neighboring WDM 
channel must be separated by a certain minimum frequency offset to be 
adequately rejected. 

[0046] The values of Re[£ 5 (/)] and Imfof/)] within the digital signal 

processor are the same as the detected optical powers that would be 
observed in the two arms of a conventional phase and quadrature homodyne 
detection system, such as with the QPSK receiver disclosed in "Linewidth 
requirements for optical synchronous detection systems with nonnegligible 
loop delay time," by S. Norimatsu and K. Iwashita (Norimatsu). Such a 
conventional receiver needs to have the local oscillator phase locked to the 
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incoming optical signal, unlike the present invention. The digital information is 
obtained from E s (t) by applying the function of a decision circuit within the 

DSP. For example, for BPSK E s takes on values [1,-1], and the decision 
circuit function operates on Re[£j with a threshold close to zero; Im[£j may 
then be ignored. For quadrature phase shift keying (QPSK), E s takes on 
values [1+/,-1 + /,1-/,-1-/] l (i.e. two bits per symbol). Separate decisions are 
made on Rq[e s ] and Im[£ 5 ] to give the two bits of content of the symbol. 

For some applications, such as the detection of a BPSK signal, 
the entire complex envelope of the electric field of the incoming signal does 
not need to be evaluated; only one component, such as the real part of the 
complex envelope, is wanted. Examples of a component of the complex 

envelope of the electric field are Re[£ 5 (/)], Im[£ 5 (/)] and Re[^(r)e^J, where 9 

is a constant. 

[0047] FIG. 3B illustrates a simplified block diagram of a balanced 
quadrature sampling receiver 70 in a first alternate embodiment of the present 
invention. Each of the phase diverse arms is further split into two balanced 
arms. The contents of the dashed box are a four branch phase diversity 
hybrid unit 72. To perform the subtraction of detected signals associated with 
balanced detection, either two differential amplifiers 74 and 76 may be used 
followed by A/D converters 78 and 80, or four separate A/D converters 82, 84, 
86, and 88 can be used and the balanced detection result obtained by 
subtraction within the DSP. 

[0048] If the phase shift imposed by the hybrid is not 90°, but given by 

e l ° , the quadrature sampling process can still be applied by replacing 
equation 6 with 

{beat term l)- cos &(beat term \)\ 



(beat term l) 



+ / 
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Only when 0 is close to 0 or tt (180°), does the quadrature detection scheme 
fail completely. 

Comparison of Quadrature Sampling With Prior Art 

[0049] Techniques such as phase switching and phase diversity have 
been developed to implement coherent detection with a local oscillator close 
to the signal's optical frequency but not phase locked to it. Phase switching 
has been demonstrated only with the differential phase shift keying (DPSK) 
modulation format (see "1 Gbit/s zero-IF DPSK coherent optical system using 
a single photodetector" by J.M. Kahn). The phase of the signal is modulated 
at the transmitter at a frequency equal to the symbol rate. At the coherent 
receiver, this modulation is removed by a DPSK discriminator, which does not 
depend on the phase difference between the signal and the local oscillator. 
Therefore, there is no need to phase lock the LO to the signal. This existing 
method has the disadvantage that the extra phase modulation broadens the 
optical signal, so that the same bandwidth detector is needed as for 
heterodyne detection. Additionally, the broadening of the optical signal makes 
the optical signal more subject to fiber propagation impairments. 
[0050] In another existing method as disclosed in "Phase- and 
polarization-diversity coherent optical techniques" by L.G. Kazovsky 
(Kazovsky 1), phase diversity detection, a 90° hybrid is used to obtain inphase 
and quadrature components. These two signals are combined to give a 
single output that goes to a decision circuit. The exact method of combining 
the two signals depends on the modulation format. For example, for 
amplitude shift keying (ASK) the inphase and quadrature signals are squared 
by analog components and then summed together as disclosed in "Wide- 
linewidth phase diversity homodyne receivers" by L.G. Kazovsky (Kazovsky 
2). The result again does not depend on the phase difference between the 
signal and the local oscillator. Phase diversity detection only works with ASK, 
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DPSK and FSK modulation formats. Thus, the existing methods cannot use 
the BPSK and QPSK formats, which provide the best sensitivity. 
[0051] Another variety of phase diversity which has been proposed is a 
double stage phase diversity detection (DSPD) method disclosed in "Double- 
stage phase-diversity optical receiver: Analysis and experimental confirmation 
of the principle" by T. Okoshi and S. Yamashita (Okoshi). The inphase and 
quadrature signals are each multiplied by a sine wave and then summed, so 
as to generate a virtual IF signal, from which information can be recovered in 
the same way as for a standard heterodyne IF signal. This method has the 
advantage that a low bandwidth detector can be used, as for homodyne 
detection, but it has the disadvantage that the subsequent components need 
to work at high IF frequencies. 

Polarization Management 
Existing Methods of Polarization Tracking 

[0052] The local oscillator beats with the optical signal only if it is to 
some extent aligned in polarization with the signal. As discussed above, it 
has been assumed that the states of polarization (SOPs) are aligned. When 
the SOPs are orthogonal, the beat term of equations 2 and 3 is zero and 
coherent detection is not possible. The SOP of the local oscillator may be 
fixed, but the signal's SOP typically varies over time because of small 
changes in the environmental conditions of the long transmission fiber link. 
Although these changes are slow, they cannot be predicted. Therefore, a 
field deployable coherent receiver requires a mechanism to match the local 
oscillator SOP with the signal SOP (i.e., to avoid the condition where the local 
oscillator is orthogonal to the signal). There are three ways to do this: 
adaptive polarization control, polarization scrambling, and polarization 
diversity. 

[0053] In polarization control as disclosed in "Comparison of 
polarization handling methods in coherent optical systems" by R. Noe et al. 
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(Noe) and "Progress towards the field deployment of coherent optical fiber 
systems" by M.C. Brain et al. (Brain), either the signal or the local oscillator 
passes through a polarization controller, typically comprising a series of 
waveplates. These waveplates are continuously adjusted under 
microprocessor control so that the SOPs of the signal and LO are aligned. 
The control of polarization must be endless (i.e., the polarization controller 
does not reach a certain state and can go no farther). 

[0054] To achieve polarization scrambling, which is described in 
"Polarization switching techniques for coherent optical communications by 
I.M.I. Habbab and LJ. Cimini (Habbab) and Noe, an extra modulation element 
is included in the transmitter, which changes the SOP of the signal 
substantially within one symbol period. At the receiver, the coherent beat 
term is effectively an average of all the relative SOPs, aligned and orthogonal, 
during the bit period, thus avoiding persistent orthogonality. Polarization 
scrambling suffers from the disadvantage of utilizing an expensive scrambler 
element in the transmitter. In addition, the act of scrambling broadens the 
linewidth of the signal, which limits the density of WDM channels and makes 
the signal more susceptible to fiber propagation impairments. 
[0055] FIG. 4A is a simplified block diagram of an existing configuration 
91 for polarization diversity detection. This solution is described in Noe. The 
signal 90 and local oscillator 92 are each divided into two paths (in addition to 
any splitting of paths for balanced detection or phase and quadrature 
detection). The splitting of the LO by a polarization splitter 94 is such that its 
SOP in the two paths is orthogonal (e.g., horizontal in the top path and vertical 
in the lower path). The SOP of the signal is the same in the two paths. There 
are two separate coherent receivers (photodetectors 96 and 98) in the two 
paths, and their outputs are summed together after demodulation of the IF, by 
demodulators 100 and 102. Then the summed result is passed to the 
decision circuit. Within this configuration, immunity to the varying incoming 
signal SOP is achieved because in the situation where one path has the LO 
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and signal SOPs orthogonal, they are aligned in the other path. The variable 
gain amplifiers in the two paths are used to avoid a variation of the peak-to- 
peak magnitude of the summed signal due to changes in the incoming signal 
SOP, as discussed in "Signal processing in an optical polarization diversity for 
560-Mbit/s ASK heterodyne detection" by B. Enning et al. (Enning). In 
addition, with polarization diversity, it is possible for the incoming signal SOP 
to vary in a manner which does not cause the relative power aligned with the 
LO in the two arms to change, but it is equivalent to the phase of the signal 
relative to the LO changing continuously (beyond 2n) in one arm, while the 
phase of the signal is fixed with respect to the LO in the other arm. However, 
for homodyne detection, it is necessary to have zero phase difference 
between the signal and local oscillator (i.e., phase locking) in both arms of the 
polarization diverse configuration. Therefore, polarization diversity cannot be 
used with homodyne detection. U.S. Patent No. 5,322,258 to Tsuchima 
(Tsushima) disclosed using an electro-optic phase modulator in one arm of 
the LO to ensure that the phase is matched, but in fact electro-optic 
modulators provide only a limited range of phase shift, and therefore does not 
provide endless polarization tracking. 

[0056] The present invention may utilize a modification of this 
polarization diversity process. As discussed above, the signal and LO 
envelopes have been assigned complex scalar variables E s (t) and E LO , The 
polarization nature is included by multiplying these scalar quantities by a 
Jones unit vector, so E s (t) becomes E s (t)p s and E L0 becomes E LO p LQ . 
The use of Jones vectors to represent polarization states is summarized in 
Appendix A. The result of coherent beating that appeared previously in 
equations 2 and 3 becomes: 

totow = Re[£ f (^ 

Ps Plo* is 1 when the L0 and S'gna' SOPs are aligned, and 0 when they are 
orthogonal. The LO has mutually orthogonal SOPs in the two polarization 
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diversity paths, which may be represented as the (real) Jones unit vectors x 
and y . The beat terms in the two paths are: 

beat term x = ^(l^e^^p.i] 

beat term y = Re[£ J (/)£^*e' fe "^ )r p s .y 

The issue described above that prevents polarization diversity being used with 
homodyne detection may be summarized as follows: p s .x and p s .y are 

complex quantities whose phase can vary independently over time. With 
homodyne detection, the function of the phase locked loop on the local 
oscillator is to adjust the phase of E L0 to compensate for any phase changes 
in the incoming signal. However, it is not possible to keep constant both 
E L0 *Ps* and E L0 *p s $. 



Polarization Tracking Using the Present Invention 

[0057] In the preferred embodiment of the present invention, 
polarization tracking is achieved by adding polarization diversity to the phase 
diversity (90° hybrid arrangement) discussed above. FIG. 5 is a simplified 
block diagram illustrating a polarization tracking system 131 in the preferred 
embodiment of the present invention. This configuration includes four 
photodetectors 110, 112, 114, and 116 (assuming single ended detection) 
and four A/D converters 120, 122, 124, 126, going to the DSP 130. If a LO 
132 is divided into polarizations x and y , the four beat terms are: 

beat term l x = Rq[e s V^^p s i] (9a) 

beat term \ y = Rc[£ f (/)£ w *e , ^^>p i .yJ (9b) 

beat term 2 X = lm[£,(/)£ La % /fe -^ o)/ p s .xj (9c) 

beat term 2 y = Im^^/^V^'^^p.-yJ (9d) 

[0058] In the case discussed above where the polarization behavior 
was ignored, quadrature sampling was utilized by forming complex numbers 
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from the two inputs to the DSP and processing them according to equation 6. 
With the polarization diversity configuration, a Jones vector is formed from the 
four inputs to the DSP, and the signal electric field is calculated from: 



7 -i(a s -G> LO )t 



r (beat term \ x ) + i(beat term 2 X ) 
^(beat term 1 v )+ iipeat term 2 y )j 



Ps (10) 



To use equation 10 it is necessary to know p s . This quantity can be 
estimated from: 

Via 



1 



1 + |*| 2 



where: 

[beat term 1 y )+ i[beat term 2 v ) 
(beat term \ x )+i{beat term 2 X ) 

The time average is preferably calculated over many bit periods to average 
out any additive amplified spontaneous emission noise. However, p s should 

be reevaluated sufficiently frequently so the changes in SOP of the incoming 
signal are tracked. 

[0059] With the present invention, this novel polarization diversity 
method is able to operate within the scenario that could not be tracked when 
using homodyne detection with existing methods. If p s .x and p s .y evolve in 

phase differently from one another, then equation 10 may still be applied. In 
such a circumstance, it will involve multiplying the top Jones vector element 
by a different phase factor from the bottom vector element. Because it is a 
mathematical multiplication that is done within the DSP 130, there is no 
constraint that the phase factor cannot be arbitrary as with an existing electro- 
optic phase modulator. Thus the present invention may be used with, for 
example, BPSK and QPSK modulation formats and provides the same 
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sensitivity as for homodyne detection (i.e., the best possible sensitivity of any 
modulation format). 

[0060] The application of the polarization diverse process of FIG. 4A 
provides more than just a complex representation of the signal that is 
calculated within the DSP 130, but additionally, it is a Jones vector 
representation that is obtained, which contains all the polarization information. 
The Jones vector E s (/) is given by: 

e ~'^ A ~ 0LO ( {beat term 1 x ) + i(beat term 2 x f\ 
8 ~ e lo * [ i beat term 1 v ) + term 2 v )> ^ ^ 



Recovery of Signals Using Other Phase-Polarization Combinations 

[0061] The passive unit 54 discussed above combines the signal and 
local oscillator into four different arms. The SOPs of the local oscillator in two 
arms are orthogonal relative to the other two arms (polarizations x and y). 
Thus, a pair of arms having the same SOP also have phases that are 90° 
apart. The Jones vectors of the LO in the four arms are: 













a) 






C) 


J J 



(12) 



These four Jones vectors can be considered to be orthogonal with respect to 
one another. 

[0062] The signal SOP and phase in the four arms is assumed to be 
the same. The relative phase of the local oscillator to the signal is particularly 
important in the quadrature sampling process. For example, quadrature 
sampling may be used if the local oscillator has the same SOP and phase in 
the four arms and the optical signal has the four Jones vector above. 
However, in discussions below, the LO polarization/phase is assumed to be 
different for the four arms and the signal polarization/phase the same for the 
four arms. 
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[0063] The signal may be recovered even if the LO Jones vector does 
not take on the four mutually orthogonal values listed above, although the 
best result is obtained when these mutually orthogonal values are used. In a 
typical detection subsystem, noise is accumulated after photodetection, such 
as from thermal noise in the photodetector and digitization noise in the A/D 
converter. When the LO Jones vectors are not optimal, the effective amount 
of post-detection noise is increased. 

[0064] Following the notation that the x-component of a Jones vector is 
denoted by adding suffix x, etc., with the Jones vectors of the LO in the four 
arms is p, , p 2 , p 3 and p 4 , and the corresponding photodetector outputs are 
beat term 1 beat term 4, then the signal can be calculated from: 









fRe[/>,,] 


Im[£ LT ] 


Re 


P\y, 


Im 


P\y, 




-l 


' beat term O 


Im 






e -i(o) s -m LO )i 
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Jjeat term 4 ; 



(13) 



[0065] Equation 11 is a special case of equation 13 for p, to p 4 given 
by equation 12. The four p, Jones vectors must be distinct from one another. 
If one of the p, is equal to another, or -1 multiplied by another (180° phase 
shift), then the 4x4 matrix in equation 13 cannot be inverted. Therefore, E s (t) 
cannot be determined. 

[0066] Thus, the quadrature sampling process may be used to 
determine the amplitude, phase, and polarization information of a signal. An 
apparatus is used which contains a local oscillator and four independent 
photodetectors. Each photodetector is exposed to a sum of signal light and 
local oscillator light. The optical phase of the LO compared to the signal or 
the state of polarization of the LO compared to the signal must be distinct 
.between the inputs to the four photodetectors. Specifically, no two 
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photodetectors must see substantially the same relative phase and the same 
relative SOP. (A phase difference of 180° is considered to be the same as 0°. 
The requirement is that the four photodetectors see distinct Jones vectors of 
the LO compared to the signal.) Two Jones vectors p, and p 2 are defined as 
being not distinct if there is a real number K such that p,=A:p 2 . An 
apparatus can be used that employs more than four photodetectors, if it is 
possible to select four photodetectors from the total that have distinct Jones 
vectors of the LO relative to the signal. 

[0067] In the preferred embodiment of the present invention, the local 
oscillator laser is usually located in the vicinity of the coherent receiver (i.e. it 
is "local"). However, in alternate embodiments of the present invention, the 
local oscillator may be remotely located from the coherent receiver. 
[0068] The replicas of the local oscillator that fall on the multiple 
photodetectors used in quadrature sampled detection are typically obtained 
by dividing the light from a single local oscillator laser, but in fact they can be 
obtained from many local oscillator lasers that are phase locked to one 
another. In addition, the local oscillator can be derived from the incoming 
signal by stripping the modulation sidebands from the signal. 
[0069] Another way that the quadrature sampling process may be used 
to obtain a representation of the signal, including its polarization information, 
employs two different local oscillators having different optical frequencies (see 
FIG. 4B). One of the LOs is split into two paths with different phases, but with 
the same SOP. The other LO is again split into two paths having different 
phases and the same SOP, but the SOP of the second pair of paths is close 
to orthogonal to the SOP of the first pair of paths. The Jones vector of the 
signal can be obtained by using a version of equation 11. If the first pair of 
paths has SOP x and LO optical frequency co LOx , and the second pair of 
paths has y and co LOyy then 
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".W=- L T 



Z,0 



f e-ifa-vu^ftbeai term lj+ /(tea/ term 2 J)' 
c^^^((&efl/ term 1 v )+/(^/ ferai 2,^ 



In this configuration, it is possible to use two different local oscillator 
frequencies because one frequency is used for each polarization state. 
[0070] It is possible to use the polarization management features 
described here without using quadrature sampling. For example homodyne 
detection can be used with a polarization and phase diversity configuration 
whose outputs are digitized and processed by a DSP. In this case equations 
10, 11 or 13 can be applied with o) s -co LO =0. Alternatively, conventional 

heterodyne detection can be used in conjunction with polarization diversity. 
The use of the DSP provides the same advantage described previously. 
Specifically, the incoming SOP can effectively be tracked endlessly even 
though homodyne detection is used, because the DSP is able to apply an 
arbitrary phase shift. 

Recovery of Polarization Multiplexed Signals 

[0071] The polarization tracking system 131 of FIG. 5 may be utilized to 
demultiplex two polarization multiplexed signals. Jones vector manipulation 
within the DSP may be used to emulate the effect of inserting a polarizer in 
the optical signal path. In general, if the input to a polarizer is E s p s and the 

polarizer has maximum transmission state p po , , then the electric field at the 

output of the polarizer is ^(ps-Ppoi'jiPpoii assuming no excess loss. The 

value of the electric field passing through the polarizer may be derived by 
using the following variant of equation 10: 

7 -i(o) s -o) LO )t r (p eai term , j(p eat term 2 x y\ # 
^(beat term \ y )+ i(beat term 2 y )j Ppo1 



£y(')PsPpol =■ 



(14) 
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[0072] Polarization multiplexing refers to a configuration where two 
transmitters of similar optical frequency are combined with polarization states 
that are nearly orthogonal. Polarization multiplexing is advantageous 
because it doubles the information carried by a WDM channel slot without 
doubling the spectral bandwidth occupied by the channel. In the situation 
where channels are launched orthogonal, the condition may be broken when 
the signals arrive at the receiver because of polarization dependent loss in the 
transmission link. In the preferred embodiment of the present invention, a 
polarization demultiplexer function is implemented that does not require the 
channels to be orthogonal. At the receive end of a conventional transmission 
system carrying polarization multiplexed signals, the two channels are 
separated (demultiplexed), usually by a polarizing element that passes one 
channel and extinguishes the other. 

[0073] The electric field of the multiplexed channels, A and S, may be 
written as: 

Re[^(/>'^'p sA+ £ jS (/>'^'p sB 
p sA and p sB are the Jones unit vectors of the SOPs of A and S. To recover 
channel A, the signal must be effectively passed through a polarizer oriented 
to be orthogonal to channel B, that is p sB *. Channel A is recovered by 
applying equation 14: 

e -'{®sA-®Lo)t ( {beat term \ x )+ i(beat term 2 X )\ „ 

E ° A ' h itt*sv>~ — — >?)r (15) 

PsA -PsB 1 * is nearly 1, given that A and B are close to orthogonal, and can, 
therefore, be ignored. In a similar manner, to recover channel B, a polarizer 
p sA * is utilized: 

7 -i(a) sB -o) LO )t f (p eQt (erm + i(p eQt term 2 x )\ ^ 



SB 



, v e infill i x j ~r lyutziAi i&r rri *» x ) A j* 

= E LO % B .pJ* ){(beat term 1 y ) + {beat term 2 y )) PsA 

(16) 
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[0074] 



The polarization multiplexed channels may be separated even if 



they are not perfectly orthogonal. They can be separated without the use of 
any extra hardware. The same polarization diversity tracking system 131 may 
be employed. To use equations 15 and 16, p sA and p sB must be known. 

p sA and p sB may be determined by an adaptive process which explores all 
of the polarization space. When the value of p sB is close to the correct value, 
it is possible to recover channel A recognizably. The bit error rate of A may 
then be used as a metric to obtain the exact value of p sB . When p sA and 

p sB are known correctly, then each signal may be recovered without crosstalk 

from the other signal. p sA and p sB must be allowed to track the slow variation 

in incoming SOPs due to the environmental disturbances experienced by the 
fiber link. The process of setting the orientation of a real (physical) 
polarization demultiplexer must also be conducted by trial and error and is 
relatively slow. The present invention provides the advantage that the 
iteration speed is determined by the computation time within the DSP, and not 
by the reaction time of any polarization control hardware. 

Obtaining Signals Using Quadrature Sampling By Analog Signal Processing 
[0075] The present invention, as discussed above, utilizes digital signal 
processing to obtain the complex envelope of the signal. Referring to the 
basic (single polarization) apparatus of FIG. 3A, the outputs of the two 
photodetectors are digitized by the A/D converters 58 and 60, and the 
mathematics of equations 7 is implemented within the DSP 36 as: 

Re K (')] = T~—\ [ cos (ta -°>Lo)t- <f>\beat term l) + s\x\((co s - co LO > - (/>\beat term 2)] 




(7a) 



Im[£ s (/)] = 




[- s\n((co s - co LO )t - </>\beat term l) + cos((o) s - co LO )t - (f>\beat term 2)] 



(7b) 
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[0076] It is advantageous to use a DSP for this task because of the 
overall flexibility of the DSP and because the complex representation of the 
signal is then available for further signal processing. 

[0077] In an alternate embodiment of the present invention, the basic 
quadrature sampling equations 7 may be implemented using analog signal 
processing elements. For example, this alternate embodiment may be 
employed in a scenario where the incoming signal has not been distorted by 
fiber propagation and the analog components cost less then the DSP 
components. As with the DSP version, the optical local oscillator may be 
close to the signal in optical frequency without being phase locked to it. 
[0078] FIG. 6 is a simplified block diagram illustrating quadrature 
sampling utilizing analog signal processing in a first alternate embodiment of 
the present invention. This configuration can be used to detect a BPSK 
encoded signal, where only the real part of the envelope of the electric field 
has to be recovered. An electrical oscillator 202 is split with one part 
undergoing a 90° phase shift, so as to generate sine and cosine signals. An 
analog multiplier 204 component may be a double balanced mixer. The 
summer component 210 performs the summation in the right hand side of 
equation 7a, and so produces an output representative of the real part of the 
envelope of the electric field of the incoming signal. The electrical oscillator 
must be phase locked to the difference frequency between the optical signal 
208 and optical local oscillator 206 so as to generate an output at a 
baseband. The circuitry to achieve phase locking is not shown, but it may be 
done by the Costas loop described in Norimatsu, for example. 
[0079] This apparatus is similar to the double stage phase diversity 
scheme disclosed by Okoshi. It is different in that the electrical oscillator 
frequency is equal to the difference frequency between the signal and LO so 
as to generate an output at baseband. With double stage phase diversity 
detection (DSPD), the electrical oscillator frequency is higher than the 
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difference frequency, and the output of the summation is a virtual IF signal 
which requires another demodulation stage to be converted to baseband. 

Compensation of Impairments by DSP 
[0080] The quadrature sampling process provides a complete 
representation of the optical signal (i.e., amplitude, phase, and state of 
polarization). All other parameters may be derived from this basic 
information. With this complete representation of the signal, the result can be 
calculated for any deterministic physical process that happens in the 
transmission optical fiber or the terminal electronics, provided the calculation 
is within the computation ability of the DSP. Any deterministic impairment can 
be reversed by a calculation within the DSP. The term "deterministic process" 
does not include the addition of noise, or the subtraction of added noise. It is 
not possible to fully reverse all impairments using an equalizer (DSP or ASP) 
after direct detection, although such an equalizer can improve the signal. The 

result of the direct detection operation is |^(/)| 2 , and the phase and 

polarization information has been discarded. 

[0081] Some of the fiber propagation impairments affecting a single 
WDM channel that can be corrected are chromatic dispersion (CD), 
polarization mode dispersion (PMD), multipath interference (MPI), and self 
phase modulation (SPM). To apply a correction for a particular impairment, a 
mathematical description of the distortion must be determined. Next, a 
mathematical formula for the inverse to the impairment must be derived. The 
established methods of DSP are utilized to convert the inverse function into 
an algorithm that can be implemented in the digital signal processor. 
[0082] Oftentimes, although the behavior of the impairment is 
completely understood, there may be one or more parameters that are not 
known. For example, to compensate for the chromatic dispersion of a section 
of optical fiber, the length of the fiber must be known. In such cases, the 
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missing parameter can be set adaptively based on the metric of lowest bit 
error rate. 

[0083] Any of the digital signal processing operations described here 
can also be applied if the coherent detection subsystem does not employ 
quadrature sampling, i.e., if it uses homodyne or conventional heterodyne 
detection followed by AID conversion. 

Chromatic Dispersion 

[0084] For chromatic dispersion (CD) on an optical signal, the CD of a 
section of fiber is described by the 2 nd order group delay coefficient J3 2 and 
the fiber length L . This is disclosed in "Nonlinear fiber optics" by G. Agrawal 
(Agrawal 1). If the electric field envelope at the input to the fiber is E in (t) t 

then the Fourier transform is denoted by E in {a>), and similarly for the output 
field E out (t). Ignoring the effect of fiber loss, the impact of chromatic 
dispersion alone is: 

E 0 M=E m ^y LML 

Inverting this relationship results in: 

%M-K^Y ¥imLL d7) 

[0085] Equation 17 deals with the Fourier transform of the signal, i.e. it 
expresses a linear filter relationship. Denoting the filter function by J{p) 



f{a>)=e ^ 



-i\P 2 a> 2 L 



and its inverse Fourier transform /(/) can be calculated. Then: 

E in (t) = E 0Ut (t)®f(t) (18) 

(® denotes the convolution operation.) Equation 18 may be applied by the 
DSP, and, in principle, compensates perfectly for the chromatic dispersion of 
the fiber section. The DSP can convolve only a finite length vector /(/), and 
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so it has to be truncated according to the computation ability of the DSP. /(/) 
takes the form of a resonance with high magnitude points close to / = 0 , so 
the truncation should not lead to a large error. If p 2 L of the link is not known, 
it can be found adaptively, such as by updating the vector /(/) by trial and 
error to obtain the best result. 

Polarization Mode Dispersion 

[0086] Referencing polarization mode dispersion, it is disclosed in 
"Systems considerations for polarization-mode dispersions" by F.P. Kapron 
(Kapron), an optical component has polarization mode dispersion (PMD) if 
light travels faster in one SOP (the fast axis) than in the orthogonal SOP (the 
slow axis). A long section of optical fiber usually exhibits strong mode 
coupling behavior, which means that the orientation of the fast and slow axes 
varies with distance. In this situation, the PMD varies with time due to small 
changes in the environmental conditions of the fiber. The PMD in the strong 
mode coupling regime, at any point in time, is described by a pair of mutually 
orthogonal principal SOPs, a first order coefficient and a second order 
coefficient. 

[0087] Oftentimes, most of the system penalty comes from first order 
PMD. If the input to a section of fiber is E in (t)p ln (constant in SOP with time), 

the output is E out (/) (not necessarily having constant SOP with time). In 

addition, the principal states have Jones vectors p PlMD and p P | V1D t , and the 
first order PMD is r , then ignoring the SOP transformation of the fiber section 
and ignoring the fiber loss, the impact of the first order PMD is: 

E out (0 = E in (tjp in .p pmd* )P PMD + E in (/ - r)(p in .p pMD t# Jfe PMD f (1 9) 
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Equation 19 indicates that the signal is separated in two according to how 
much of the signal lies in the two principal states and one state is retarded in 
time by r while the other is left alone. Equation 19 is reversed by: 

£/*(0= E out(0-PPMD*^ -Ppmd**) 

(20) 

[0088] This relationship may be implemented by the DSP so as to 
compensate for the first order PMD. r is typically not a whole multiple of the 
sampling interval of the A/D. Therefore, it is necessary to perform 
interpolation to obtain both E out (/) and E out (f + r). p PlMD and r are not 

known initially to the DSP and must be found adaptively. These values vary 
with time and, therefore, the DSP must track the real values. 
[0089] PMD is one of the impairments that can be partially 
compensated by electrical equalization in conjunction with direct detection. 
The method discussed above, however, in principle completely reverses the 
effect of PMD in this situation. 

Multipath Interference 

[0090] Multipath interference occurs when an optical signal is split into 
two or more paths having different physical lengths and then recombined. 
Usually one path (i.e., the main path) carries a much stronger signal than the 
others, but the power in the "echoes" arriving via the other path degrades the 
signal that has traveled through the main path. FIG. 7A is a simplified block 
diagram illustrating a subsystem 300 that contributes MPI, containing a pair of 
optical amplifiers 302 and 304 connected in parallel. The two optical 
amplifiers have different passbands. FIG. 7B is a graphical representation of 
an associated gain spectrum for FIG. 7A. The configuration in FIG. 7A is 
used to produce amplification over an extended optical bandwidth. However, 
it is not possible to operate over a continuous bandwidth because there is an 
unusable region in between the two passbands where the MPI is too great. At 
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a wavelength in the dead zone, typically one amplifier has more gain, called 
G, , but the gain of the other amplifier, G 2 , is sufficiently large that MPI is a 
problem. The impact of this situation of two path MPI on the signal electric 
field is (ignoring the effect of the optical polarization): 

Eout if) = 4G x E in (/) + Jg^-AV"* E in (/ - x MPl ) (21 ) 

where r MPl is the difference in time delay of the two paths. 
[0091] To compensate for the MPI within the DSP, the relationship of 
equation 21 must be inverted. The easiest way to accomplish this inversion is 
in the z-transform domain. The z-transform of equation 21 is: 

E out{z) = 



<Jg~\ + <Jfye~ ia>sTMPI z Tsamp 



EM 



where r samp is the sampling interval of the A/D converter and it is assumed 
that x MPl is a whole multiple of r samp . The MPI impairment is reversed by: 

Em W = ! — E out (r) (22) 

7^7 + y[G^e- i0 ° TM '' 1 z Tsamp 
[0092] MPI compensation can be achieved by implementing a digital 
filter within the DSP. As explained in Lyons, the output y{n) of a general 
recursive digital filter is computed from the input x(n) by: 

CO CO 

The transfer function of equation 22 may be obtained from a digital filter using 
these coefficients: 

oo= ir 

a k = 0 k > 0 
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b k = 0 otherwise 
r MPI and G 2 /G, can be found adaptively by the DSP given that they are not 
known initially. 

[0093] The above example provides compensation of a two path MPI. 
It is possible to compensate for more paths by adding more terms to the 
recursive filter algorithm, though for a large number of MPI paths it is 
computationally difficult to implement. 

Adaptive Equalization 

[0094] In addition to compensating for known physical impairments, the 
DSP may improve the signal without determining the cause of the distortion. 
Adaptive equalization is used in radio transmission systems, and similar 
algorithms may also be used after coherent optical detection (See Proakis, 
chapters 10 and 11). Each equalizer type relies on a set of parameters to 
customize its function. These parameters may be determined during normal 
operation based on a metric of the quality of the received signal, or during a 
training phase when a known sequence is sent through the system. The least 
mean squares algorithm is often applied as the control algorithm to obtain the 
parameter set. Examples of adaptive equalization methods are feedforward 
equalization-decision feedback equalization and maximum likelihood 
sequence estimation. 



Optical Filter Control 

[0095] When coherent detection is used, it is equivalent to inserting a 
narrow optical filter centered around the local oscillator wavelength. The 
passband shape of the filter is symmetric, and is given by unfolding around 
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o) = 0 the baseband response of the detector + amplifier chain + other 
components prior to the decision circuit (in the present invention, the AID 
converter + DSP). Within the preferred embodiment of the present invention, 
the effective filter shape may be tailored within the DSP. An adaptive filter 
may be incorporated in the DSP first to undo any unwanted features in the 
frequency response of the detector etc. 

[0096] The ability to create an arbitrary filter shape is advantageous in 
optical signal transmissions. Much attention is paid to the shape of the WDM 
demux filter in dense WDM system design. The filter must have a flat 
amplitude response and a low group delay ripple within the spectral window 
occupied by the signal, and steep walls outside the window to reject 
neighboring channels. There are also applications where the optical filter 
shape is engineered to improve the bit error rate of the signal passing through 
it, such as to improve the tolerance to SPM of an NRZ signal [see "Interplay of 
Fiber Non-Linearity and Optical Filtering in Ultra-Dense WDM" by I. 
Lyubomirsky et al. (Lyubomirsky)] and to reject one of the sidebands of an 
NRZ signal [see "0.8 bit/s Hz of Information Spectral Density by Vestigial 
Sideband Filtering of 42.66 Gb/s NRZ by W. Idler et al. (Idler)]. In addition, 
the filter shape may be set to be different for different signal types (e.g., wide 
for high symbol rate signals, and having steep skirts for dense WDM signals). 

[0097] For example, assume that the desired filter shape is f{co -cd lo ). 

The filter shape does not have to be symmetric about co-(o LO = 0 , as it would 

if an analog signal processing stage were used. The constraint does not 
apply because the DSP is processing complex electric field values. Assuming 
that the detector + amp chain effectively has a flat frequency response, the 
optical filter is implemented by: 

E sJilt (t) = E s {t)®f(t) 

where /(/) is the inverse Fourier transform of /(&). 
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Subtraction of Signal Power term When Using Single ended Detection 

[0098] Recalling that the optical power in one arm of a coherent 
detection apparatus is given by equation 1: 

/>, =\E s (tf +\E LO \ 2 + 2Re[E s ( t )E L oe^-<°<-o)>} (1) 

The local oscillator power \E L0 \ 2 is constant, so the detected power is equal 

to the beat term only if the local oscillator power is considerably larger than 

the signal power. Differential detection allows the \E s (t} 2 term to be 

subtracted. However, it is preferred to utilize single ended detection because 
it saves the cost of a second photodetector. 

[0099] It is possible to deduce E s (t) from the two phase diverse arms 
of a quadrature sampling receiver, even if the LO power is not much larger 
than the signal power. A first estimate of the signal envelope E sestl (t) is 

made using the method for quadrature sampling described above. This first 
estimate will be related to the true value E s (t) as follows 

E s es« (<) = E s (t)+ \E g (if J±Le-**-*<°> (23) 

The proportional deviation of E sestl (t) from E s (t) (the second term in equation 

23) has magnitude . Then a second estimate can be made as 

V2|£, 0 | 

follows: 

E s esa it) = E s estx (0 - \E S esli (tf _i±i_ e -''( c '.v-®/.o> (24 ) 

2Elo 

E sesl2 is closer to E s than E sestX provided that the power of the LO is greater 
than that of the signal. In actuality, E sest2 is related to E s by: 
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E sest2 {t)=E s (t)~ 



mm 2 _ <m(tf c -nu-a>, n )< 

2\E LO \ 2 2E LO * 2 



c -iU-a>,n)t 

4E lo*\Elo\ 2 




The proportional deviation of E sestl (t) from E s (t) now has magnitude 



which is smaller than it was for E sestX . Repeated iterations of equation 24 
yield an estimate of E s that is closer to the actual value. 

Comparison with Prior Art on Impairment Compensation 



demonstrated with heterodyne coherent detection, as disclosed in "Chromatic 
dispersion compensation in coherent optical communications" by K. Iwashita 
et al. (Iwashita). The compensation is done by inserting an analog signal 
processing component in the IF path which has the correct group delay vs. 
frequency curve in the IF band to reverse the group delay vs. frequency the 
signal experienced in the optical band. Additionally, the double stage phase 
diversity method described above generates a virtual IF signal from baseband 
inphase and quadrature signals, and chromatic dispersion can be 
compensated by ASP in the IF band in the same way as for standard 
heterodyne detection (see Okoshi). This proposal has the advantage that a 
low bandwidth detector can be used, as for homodyne detection, but it has the 
disadvantage that the following components need to work at high IF 
frequencies. K. Yonenaga in "Dispersion compensation for homodyne 
detection systems using a 10-Gb/s optical PSK-VSB signal" (Yonenaga) 
discloses compensating for CD after homodyne detection, but disclosed using 
a vestigial sideband (VSB) modulation format. However, VSB formats require 
extra complexity in the transmitter and suffer from the disadvantage of having 
a worse sensitivity than standard modulation. 



[0100] 



Electronic compensation for chromatic dispersion has been 
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[0101] The present invention has many advantages over existing ASP 
processes for impairment compensation. The present invention, unlike 
existing ASP systems, works with homodyne as well as heterodyne detection. 
There is no requirement to utilize components that work at frequencies higher 
than the symbol rate, the amount of chromatic dispersion can be varied under 
software control, and other impairments can be compensated which do not 
have a response that can be engineered into an ASP component. 



Crosstalk Subtraction 
[0102] The discussion above dealt with signal processing of a single 
WDM channel. However, there are impairments in a WDM transmission 
system that are driven by other WDM channels, such as linear crosstalk 
(WDM demux crosstalk), cross phase modulation (XPM) and four wave 
mixing (FWM). A WDM channel suffers from crosstalk if the quality of the 
received signal (measured by the BER in the case of a digital signal) is worse 
in the case when other WDM channels are present than when they are 
absent. The present invention can overcome these impairments. 
[0103] To adequately subtract any crosstalk from a signal, the main 
channel and the other WDM channels that contribute to the impairment are 
observed by sampled coherent detection. An initial estimate of the complex 
envelope of all the channels taking part in the crosstalk process is obtained. If 
necessary, an optical filter may be applied. A mathematical description of the 
crosstalk process is applied to calculate the crosstalk on the main channel. 
Next, the crosstalk is subtracted or reversed from the main channel. 
[0104] A coherent receiver only sees a region of the optical spectrum 
about 2X the receiver bandwidth, centered on the local oscillator wavelength. 
To make observations of many WDM channels simultaneously, it is necessary 
either to use a coherent receiver with sufficiently wide bandwidth to observe 
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several WDM channels at once, or to use several sampled coherent receivers 
with different local oscillator wavelengths and combine the sampled 
description of many channels in one digital signal processor. The requirement 
on high speed digital data transfer and on computation ability of a DSP grows 
rapidly as it is asked to process more WDM channels. However, any 
impairment may be reversed with complete knowledge of the entire received 
spectrum. 

[0105] To detect two channels, centered on co sA and co sB , using the 
same receiver, equation 6 is used with different signal center frequencies and 
a filter function f(co) (having inverse Fourier transform /(/)) is applied to 
remove the other channel as shown below: 



E sA (t) = 
E s b{() = 



( e -i{0>sA-0>Lo)< * 

[(beat term \)+i(beat term 2)\ 

E LO 

( „-i(«>sB-<»Lo)t 



«/(') (25a) 



[(beat term l)+ i(beat term 2)] 



v 



<lo 



®f(t) (25b) 



Linear Crosstalk Subtraction 

[0106] The spectrum of an optical signal modulated by high bit rate 
information has a finite width. The 3dB width is at least a value equal to the 
symbol rate. In addition, the spectrum typically has "skirts" which fall off at a 
finite rate moving away from the center of the channel. In general, to 
separate two closely spaced WDM channels, a narrow optical filter is required 
which has steep walls. The DSP technology in conjunction with quadrature 
sampled coherent detection is able to synthesize an arbitrarily steep walled 
filter. However, if the channels are spaced too closely, then they will overlap. 
No matter what optical filter passband shape is chosen, if it passes one 
channel without distorting it, then it will also admit some energy from the 
adjacent channel. The crosstalk in this scenario is known as linear crosstalk, 
and the situation can be improved by using crosstalk subtraction. 
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[0107] For two WDM channels, A and B, in the presence of additive 
noise from optical amplifiers n((), the signal electric field (ignoring SOP) 
arriving at the receiver can be written as: 

[0108] A first estimate of each of the channels may be made using 
quadrature sampling, either in one receiver or in two separate receivers with 
different LOs, and applying a narrow filter having impulse response f(t), per 
equations 25. The estimates will each contain crosstalk from the other 
channel. 

E A esa if) = E A it)® /(,)+ [n(ty^ )® /(/) + (e b )® /(,) 

(26a) 

E B esa if) = E B (t) ® /(,) + (^y* )® /(/) + [e a (/>^— > )® /(,) 

(26b) 

The first term of the right hand side of equations 26a and 26b is the filtered 
version of the desired signal, the second term is noise, and the third term is 
crosstalk. The presence of crosstalk means that the bit error rate will be 
higher than it would be if the other channel were not there. The two estimates 
can be passed to a decision function, denoted by £>(..), which predicts which 
of the allowed values was transmitted. The quantized value of each channel, 
Q{ E csia) or Q{E es t b). can then be subtracted from the other channel to make a 
second estimate: 

E A es a if) = E A est] (/)- {${E B esn (f)><-rt > )® /(/) (27a) 

E B esa if) — Eq eslX if) - fe{E A est] (f )>^— > )® /(/) (27b) 
This second estimate will generally be better than the first. 
[0109] The accuracy of E Aest2 , which uses crosstalk subtraction, can be 

compared with E A€Sflt which does not utilize crosstalk subtraction. 
Substituting equations 26 into 27a provides: 
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(28) 

The function E B (t)-Q(E BestX (t)) is zero most of the time, and has a pulse 

whenever a bit error occurs based on E BestX . The noise term in equations 28 

and 26a cannot be avoided, and in fact it is the target to make the system 
noise limited. Comparing equation 28 with 26a, E Aest2 is closer than E AestX 

to E A (t)®f{t) provided that E B -Q>(E BestX ) has a lower root mean square 

(r.m.s.) deviation than E BestX . This is correct when the BER of E Bestl is lower 

than about 0.5. The channels should be spaced less than approximately the 
symbol rate before this condition is violated, so linear crosstalk subtraction 
enables a very low channel spacing to be achieved. 

[0110] Thus, the present invention has been described herein with 
reference to a particular embodiment for a particular application. Those 
having ordinary skill in the art and access to the present teachings will 
recognize additional modifications, applications and embodiments within the 
scope thereof. 

[0111] It is therefore intended by the appended claims to cover any and 
all such applications, modifications and embodiments within the scope of the 
present invention. 
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Appendix A 

Use of complex numbers to describe modulated signals 

The discussion above utilizes complex numbers to describe sine and 
cosine functions because this notation is a compact way of including the 
phase of the sine wave or cosine wave. For example the electric field is 
written in the form: 

E(t)=Kt\E s e lm \ (A1) 
where E s is a complex number. This can be expressed in terms of sines and 
cosines as: 

E(t)= Re[£jcos(*>/)- Im[^Jsin(^) 

Or if complex E s is written in terms of its magnitude and phase as: 

E s =\E s \e id * 
then A1 becomes: 

E(t) = \E s \co*{a* + O s ) 

The complex number notation is compact because the phase of the 
sine wave is stored in the phase of the complex number. 
In some places in the discussion, there appear equations like: 

beat term = Re[E s E L0 *e im \ (A2) 

E L q is the complex conjugate of E LO , meaning that every occurrence of / is 
replaced with -/, and: 

Elo =\E L o\e- ieLO 
So A2 can be rewritten as: 

beat term = \E S \\E L0 \ cos(a)t + 0 S - 9 LO ) 

The appearance of E S E L0 * in A2 means to take the phase difference 
between E s and E L0 . 
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The power of an optical wave is given by the magnitude squared of the 
complex electric field, and does not have a sinusoid time dependence. In the 
case of a field given by A1 : 



Jones vectors 



The state of polarization of an optical signal can be described by a 
Jones vector. This is a two element column vector. Each element is the 
complex envelope of the electric field, i.e. phase information included. The 
top element is the component of the field in the x-direction (horizontal) and the 
bottom element in the y-direction (vertical). In fact x and y can be an arbitrary 
pair of orthogonal directions. [See "Optics" by E. Hecht "Hecht"]. 

Some Jones vectors of familiar states of polarization are listed below. 



i 

4i 



horizontal 



vertical 



f \\ 



J_f1 

V2 



linearly polarized at 45° 



circular 



A Jones unit vector p has the property that is: 
P P* = 1 

If light polarized in SOP p, passes through a polarizer oriented in direction 



p 2 , then the electric field is scaled by pYpY . In general 0 < 



Pi P2 



<1. 



46 



5034-0001 
PATENT 



The Jones unit vector of the state orthogonal to p is denoted in the above 
discussion by p*. and 
p.p T =0 



If 



p = 



Pyj 



then 



P T = 



Px 

P X 



Px 
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